Optimal Decision Trees

We propose an Extreme Point Tabu Search (EPTS) algorithm that constructs globally optimal decision trees for classiication problems. Typically, decision tree algorithms are greedy. They optimize the misclassiication error of each decision sequentially. Our non-greedy approach minimizes the misclassiication error of all the decisions in the tree concurrently. Using Global Tree Optimization (GTO), we can optimize existing decision trees. This capability can be used in classiication and data mining applications to avoid overrtting, transfer knowledge, incorporate domain knowledge , and maintain existing decision trees. Our method works by xing the structure of the decision tree and then representing it as a set of disjunctive linear inequalities. An optimization problem is constructed that minimizes the errors within the disjunctive linear inequalities. To reduce the misclassiication error, a nonlinear error function is minimized over a polyhedral region. We show that it is suucient to restrict our search to the extreme points of the polyhedral region. A new EPTS algorithm is used to search the extreme points of the polyhedral region for an optimal solution. Promising computational results are given for both randomly generated and real-world problems.